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(57) Abstract 

An interfacing device and method for controlling and accessing information and communication services and applications are described 
(10. n. 12. 17, 18). The method relates to a common virtual canvas Unking the display space of a large, central public display (17) with 
the private work and display spaces of ambulatory, high-resolution touch-activated devices, such as lap-top controllers (10). Instead of 
actually transferring information from one lap-top controller (10) to another or to the public space, with a lap-top controller (10) the uscr^ 
are actually controlling the permission of others to access certain information transmitting the information to the set-top box (10, II, 12, 
17, 18). Therefore, the user is actually controlling the state of their collective virtual canvas by manipulating permissions which determine 
each user's view (17) of multimedia information stored in central network-based servers. 
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A LAP.TOP DEVICE AND METHOD 
FOR ACCESSING AND CONTROLLING COMMUNICATION SERVICES 

RELATED APPLICATION 
This application is a continuation of provisional U.S. Patent Application Serial 
5 No. 60/004,5 19 filed on September 29, 1995, having the same title, inventor, and 
assignee as the present application. 

FIELD OF THE INVENTION 
My invention relates to an interface device and method specifically for 
accessing and controlling information and communication services and applications. 

10 BACKGROUND OF THE INVENTION 

As the vision for a powerful National Information Infrastructure (Nil) 
comprising an interconnection of intelligent, broadband digital networks is more clearly 
defined, it becomes apparent that user interfaces to control these all-purpose networks 
will have to satisfy a number of conflicting requirements imposed by the wide range of 
15 high-quality, interactive multimedia applications. Users will need to read textual 

information from screens without fatigue, much as people do now with today's books, 
magazines and other documents. Large screens will display high-resolution, full-motion 
entenainment-quality multimedia applications, allowing users to immerse themselves 
fully in the visual imagery. But these users will need simple and easily remembered 
20 methods for controUing information access, and these methods will need to remain fairly 
constant across multiple applications. Finally, user interfaces that inherently suppon 
cooperative and collaborative enienainment, work, and play will be advantaged in the 
new information age. Therefore, a relatively small set of powerful, easy to learn, and 
consistent methods will have to be developed if users are to access a broad range of Nil 
25 services successfully. If such uniform methods are not established, there could be a 
number of tools with diverse human interfaces and methods that may curtail the 
deployment and success of the NIL 

To these ends, objectives of my invention are a new interface device and 
method for controlling information and communication services and applications on 
30 different networks. 

SUMMARY OF THE INVENTION 
My invention relates to an interface device and method for accessing and 
controlling information and communication services and applications on different 
networks. My method is based upon the concept of a common virtual canvas linking the 
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display space of a large, central public display wuh the private work and display spaces 
of ambulatory, high-resolution, touch-activated devices, such as a lap-top controllers 
Instead of actually transferring information from one lap-top controller to another or to a 
public space, with my lap-top controller the users are actually controlling the permission 
5 of others to access cenain information. By pushing information into a public space, the 
user signals a controlling set top box to permit the information to be transmitted to a 
public display, without the user actually transmitting the information to the set top box. 
Therefore, the user is actually controlling the state of their collective virtual canvas by 
manipuiatmg permissions which determine each user s view of multimedia information 
10 stored in central network-based servers. 

BRIEF DESCRIPTION OF THE DRAWINGS 
Figure 1 A depicts an illustrative embodiment of a system comprising two lap- 
top controllers linked to a set top box and a large television set in accordance with an 
aspect of my invention. 
15 Figure IB depicts an illustrative embodiment of my lap-top controller m 

accordance with an aspect of my invention. 

Figure 2 depicts the transfer of objects from the user's private space on my 
lap-top controller to a shared public view, in accordance with an aspect of my invention. 
Figure 3 depicts the lap-top controller being used to zoom into details of 
20 buttons on a dress from a home shopping application in accordance with an aspect of my 
invention. 

Figure 4 depicts two lap-top controllers at the same location used to work 
collaboratively on the shared public space in accordance with an aspect of my invention. 

Figure 5 illustratively demonstrates the transfer of multimedia directly between 
25 lap-top controllers in accordance with an aspect of my invention. 

Figure 6 illustratively demonstrates the transfer of multimedia information over 
a network using lap-top controllers in accordance with an aspect of my invention. 

Figure 7 illustratively depicts the video streamer technology used to access or 
preview video to be displayed on shared public space in accordance with an aspect of my 
30 invention. 

Figure 8 depicts illustrative embodiments of display screens for lap-top 
controllers in accordance with an aspect of my invention. 

Figure 9 depicts an illustrative embodiment of a display screen of a lap-top 
controller with a TV listings interface in accordance with an aspect of my invention. 
35 Figure 10 depicts an illustrative embodiment of a display screen for lap-top 

controllers being used to preview still frames from several TV channels before making a 
selection for programming on a large, shared, display in accordance with an aspect of my 
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invention. 

The organization and operation of the invention wjJl be better understood from 
consideration of the detailed description of the illustrative embodiments thereof, which 
follows, when taken in conjunction with the accompanying drawings. 

5 DETAn.ED DESCRIPTION OF THE INVENTION 

My invention relates to an interface device and method for controlling 
information and communication services and applications on different networks. Mv 
method is based on the concept of a common virtual canvas which links the display space 
of a large, central public display with the private work space and display space of 
10 ambulatory, high-resolution, touch-activated devices. 

My user interface system solves some immediate problems for emerging 
broadband video distribution channels, while positioning for future full-service networks 
and ultimately for the Nil. While this system is suitable for all market segments, this 
application focuses illustratively on residential applications for ease of discussion. 
15 Most users of home television services currently have a television receiver with 

a remote control device. Currently, most providers of advanced information networks 
expect that some sort of set-top box will be used to access the advanced offerings of full 
ser\ ice networks (FSNs) and that the television receiver will be the only output device 
for most of these services. Therefore, a single display device must handle technically 
20 conflicting requirements, namely displaying bright and colorful full motion entenainment 
video along with the high-resolution text and graphics that will be needed to suppon user 
interfaces for services such as video browsing and information searching.a 

I take a different approach to resolving the problem of uniform information 
access and display. Herein, I disclose using a set top box in conjunction with a new 
25 consumer interface device that would replace the remote control commonly used with 
most television receivers. This new device, which I refer to herein as a lap-top 
controller, has some aspects of the emerging personal digital assistants (PDAs) but is 
designed to be an integral pan of the FSN interface system. 

Illustratively as shown in Fig. IB and in accordance with my invention, the 
30 lap-top controller 10 comprises an infrared (IR) transceiver 1 1 and a radio frequency 
(RF) transceiver 12 for communicating with a set top box 14 which is connected to a 
shared-area large screen "public" display 15, as seen in Fig. lA. My lap-top controller 10 
also comprises a display 17. having a large area for the user to view information located 
in his or her public and private spaces. Furthermore, my lap-top controller comprises a 
35 central control processor 18 which controls and coordinates the transfer, receipt, and 
display of information into and out of the controller based upon signalling from the user 
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and other users and based upon software used to operate the lap-top controller. 

My lap-top controller is characterized as having a large display area, in display 
17 the contents of which can be manipulated by the user by providing direct physical 
input via touching or use of a stylus, and might advantageously be approximately similar 
5 m size to the children's toy; "Etch-A-Sketch". A display resolution of 75 pixels/inch or 
more would make graphics and text readable at convenient hand-held distances. The 
device is capable of displaying full color video at 30 frames per second over us entire 
surface. Although full motion video is imponant for some of the more advanced uses of 
the lap-top controller, many powerful new capabilities can be enabled with slower 
10 displays. My experience with large screen video-conferencing systems indicates that a 
display area with an 8x3 aspect ratio (twice the 4x3 aspect ratio of NTSC television) 
would be most desirable as a preferred embodiment. In addition to allowing more area to 
accommodate public and private viewing and work spaces, this wider aspect ratio would 
also allow the device to emulate a full-width keyboard if needed. 
15 My lap-top controller is lightly coupled with the shared public display 15 (and 

other private displays) by RF and IR linking technologies and by operation software 
running in the network, the set top box. and in the lap-top controller itself 

My lap-top controller is linked to the set lop box with two channels. One 
channel supports the high speed delivery of full-motion video and high resolution 
20 graphics to the controller. The other channel is a bidirectional data link. The two 

channels work coop>eratively to allow the flow of multimedia information between the 
lap-top controller and the set top box. By using the data link to control flow on the high 
speed link, both graphics and video could be accommodated. 

For user input, the lap-top conu-olier includes a touch screen over the display 
25 surface. With appropriate software in the lap-top controller, its user could use a fingenip 
(or stylus) for written input or for image manipulation on the face of the display. An 
experimentaLl research prototype of an initial version of a lap-iop controller was built 
around the Datalux Touch LCD Monitor. The 640 X 480 pixel, 256 color, AMLCD flat 
panel display is adequate for the presentation of easily readable text and moderate 
30 resolution images. It is mated with the Elo TouchSystems Accutouch® touchscreen 
which provides fingertip data access and control. 

Figure 1 A depicts an illustrative embodiment of my lap-top control system 
comprising two lap-top controllers 10- 1 and 10-2 linked to a set top box 14 and a large 
television set 15. The set top box handles the interface with the FSN and provides 
35 wireless links to the lap-top controllers 10 and a wired connection to the television set 15 
The television display is optimized for full-motion, full-color enienainmeni video and is 
esseniiaily the same as present-day TVs. This TV might have a very large ( projection t 
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display area, and might support enhanced resolution signals. The display on the lap-top 
controllers can support small video windows, but are primarily designed for high 
resolution graphics and text. 

By providing separate viewing spaces the lap-top control system can support 
5 new and novel information manipulation methods. These methods will facilitate access 
to a wide variety of information and coinmunications applications and allow multimedia 
browsing in a powerful and versatile manner. 

An outstanding aspect of my lap-top controller and method is the linking of the 
private display area of a lap-top controller with the larger, common public display area 
10 (the television set) and, under certain conditions the linking of one lap-top controller s 
display area, to the display areas of other lap-top controllers. As shown in Figure I A. the 
display areas of the TV set 15 and of each of the lap-top controllers 10 are pan of a 
single, large virtual canvas. Objects can be moved with great facility within a space and 
from one space to another in a manner that simulates moving physical objects. My lap- 
i 5 top controller 10 provides users a powerful visual representation of what is actually 
happening as they manipulate information on their screens. 

All objects that can be depicted on the surface of my lap-top controller can be 
manipulated by pushing them around on the display surface. As with using a mouse, 
objects can be selected by touching them in a specified way. They can be activated with 
20 a set of easily recalled pressure or touch sequences and can be pushed or dragged to new 
locations by simply touching them with a finger. This includes repositioning video, 
graphic, or textual data as well as activating functions that operate on this data. My 
approach can be applied even for data that has been slid onto public areas or other private 
areas on the user's display. 

^" °"<^ "^^e of operation, depicted in Figures 1 and 2. a fixed portion 20 of the 
lap-top controller's display space is used to represent the common space. Users can 
assemble some needed combination of objects in their private spaces 21 and. when 
satisfied with the result, slide them into the area of the public space. Other users in the 
same physical area (e.g. living-room) can now see the final visual result of an effort that 

30 may have taken a good deal of local manipulation to generate. Once the information is in 
the public space, other users can slide a copy into their own private spaces on their lap- 
top controllers for their personal modification. If the information represents objects (in 
both the physical and software sense of the word) then these users can add these objects 
to their stock of executable objects. 

35 In a related method, as illustratively depicted in Figure 3. the user may have a 

local display of the "common area" that can be shrunk or enlarged at will to envelop ail. 
or any part of the user's private work space. In one example, the common area could be 
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used to expand a rectangular area that occupies a small (loo small to view comfortably) 
area of the lap-top controller display. Users could push this fixed-sized box around on 
their private space, causing the common space to act as a giant magnifying glass. The 
area of the user's screen within the rectangle would blow up to fill the canvas of the 
5 shared screen. In a related and more powerful capability, the movable common-screen 
representation area on the lap-top controller could be scaled up or down on the screen bv 
stretching it with the fingers. If it is made bigger, the magnification going from lap-top 
controller to common area is reduced. If the window is made smaller, the masnifvins 
effect IS made larger. 

10 As a special case of the relationship between private and public spaces, the 

user's entire lap-top controller workspace could be dragged around on the vinual canvas 
as shown in Figure 4. If several users dragged their private spaces into the common 
space, the contents of their private spaces could be simultaneously displayed for 
comparison on the large common area screen. Their work could be easily reviewed, 

15 superimposed for comparison, or connected to form a larger common work. 

The transfer of multimedia objects between private spaces, and in conjunction 
with the public space, represents an important use of this new method. Here, in addition 
to its own private area, each lap-top controller could call up a portion of its display area 
representing the private space of another lap-top controller as shown in Figure 5. 

20 Passing information from one collaborator to another would then be as simple 

as sliding it into the other user's private space. As it is moved into this space on the local 
controller, the information would appear in the private space of the colleague's 
controller. This sharing operation is analogous to the public/private area sharing 
discussed earlier. 

25 A powerful corollary to the public/private space paradigm is that the network 

could be used to interconnect users at separate physical locations. Geographically 
separated users could then share multimedia information. For example, as shown in 
Figure 6. a user in one location could push a picture of a zoo animal into his public space 
and have it appear in the public space of a colleague (or colleagues) across the countrv . 

30 Instead of actually transferring information from one lap-top controller to another or to 
the public space, with my lap-top controller the user is actually controlling the 
permission of others to access certain information. By pushing information into the 
public space, the user signals the set top box to permit the information to be transmitted 
to the public display, without the user actually transmitting the information to the set top 

35 box. Therefore, the user is actually controlling the state of their collective virtual canvas 
by manipulating permissions which determine each user's view of multimedia 
information stored in central network-based servers. 
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In order to augment the lap-top controller's ability to access, as well as share 
multimedia information, there may advantageously be employed a multimedia play back 
mechanism, such as that discussed in U.S. Patent Application Serial No. 08/300.824, 
filed on September 2, 1994 for inventors Cruz, RohaJl, Rosenberg, and Smoot which 
5 mechanism is referred to herein as Video Streamer. Video Streamer is a way to represent 
and display large amounts of video information in a two dimensional space as shown in 
Figure 7. Here, video frames 30 are sampled, at intervals, out of a stored video clip, and 
arranged along a time-line like an overlapping deck of playmg cards (with the edges 
exposed). 

10 In order to use the interface, one assumes that a streamed version of, for 

instance, a movie one is interested in, is displayed on the lap-top controller's surface. A 
finger tip is dragged along the time axis 3 1 of the stack to locate key scenes in the movie. 
As the finger is moved, the nearest sampled still frame is displayed. A 90-niinuie movie 
could be perused in seconds, with only a small sub-sampling of actual visual information 
15 needing to be sent to the lap-top controller from a central, network- linked media server. 
If the user finds that the clip appeals to them, then the entire document could be pulled 
from storage with its play-out commencing at a selected frame. 

A Streamer time axis can also represent nonlinear temporal relationships. For 
instance, preprocessing a video stream could determine major scene changes and thus 
20 extract the more important aspects of a movie or other presentation. When the stack of 
visual information is displayed, the individual samples could be spaced by distances on 
the time line "vinual Z axis" that represented their actual occurrences. 

The "Z" axis of a Streamed output could represent parameters, other than time, 
that are related to various descriptors of the material being displayed. For instance, the 
25 individual samples could be residential homes in a continuum of increasing or decreasing 
prices. Houses which were similar in another dimension (e.g. floor space) could be 
shown in close proximity on a Streamer axis. Similarly, in a shopping application, men s 
suits could be lined up by color, material, or cost in separate Streamer stacks that were 
automatically compiled from a network data base. 
^0 Although many home entertainment applications will not require a keyboard, 

there are other applications where the flexibility and precision of a keyboard are required. 
In these cases the lap-top controller could emulate a keyboard by using the touch screen 
capability in conjunction with a graphic or photo showing the desired keys. Audible 
feedback could emulate key "clicks." The power of this approach lies in its adaptability 
35 to the task at hand. User interfaces could be drastically simplified by limiting each 
keyboard to those keys needed at each stage of an application. An example of such a 
* soft keyboard" is shown in Figure 8 where a numeric keypad 40 is provided for the 
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direct entry of TV channel numbers in a TV channel controller application. 

My method, using the lap-top controller for controlling network applications 
has considerable flexibility. One can envisage lap-top controllers with advanced 
capabilities. In addition to the basic display and touch-sensor described above, a lap-top 
controller might allow for biometric access (e.g., hand print or shape ID), or might 
provide a microphone for use with voice recognition. Additional input devices might 
include a scanner for documents or a video camera for video-conferencing. 

Application of my lap-top controller to movie browsing and to home shoppms 
has been disclosed above, but there are some other application scenarios to further 
illustrate the versatility of the interface. A near-term, but powerful, use of a lap-top 
controller is for controlling access to TV programming. As cable and hybrid fiber-coax 
systems expand from 50 to a 100 or more channels, it becomes difficult for users to 
remember what is available to them. In the simplest case, each channel could be 
graphically represented by an icon 41 depicting the station logo as shown in Figure 8. 
15 These icons could be grouped according to classes (off the air, pay TV, personal 

favorites, etc.) The user would tune his or her TV by touching a button for the class of 
programming desired, then touching the icon of the desired service provider. Figure 8 
shows **screen dumps*' from my laboratory prototype of such a user interface. The right 
screen assumes that a "FAVORITES*' button has been pressed to show a previously- 
20 designated group of channels. The user can then, for example, touch the CNN logo 4 1 to 
tune to that channel. The left screen shows how a "soft keypad" 40 could be provided to 
allow the user to tune to a channel by direct entry of its number. This example assumes 
that the user has touched first 5, then 6, to tune channel 56. 

The TV access application detailed in Figure 8 has been implemented in our 
25 experimental research prototype and researchers are able to move between several 
screens simulating moving through a Television browsing session. Researchers can 
change channels by touching station logos, or by pressing the "buttons" on a pop-up 
keyboard. 

Another powerful way to access TV channels would be to use the lap-top 
30 controller to display a program-listmg grid as shown in Figure 9 (much like those found 
in TV Guide or newspaper listings). Although listings services have been proposed and 
trialed in the past, the use of the normal TV display has severely limited the scope and 
depth of their listings. The high resolution of the lap-top controller coupled with its 
proximity would allow the user to browse the entire TV listing for a particular night. A 
35 grid that overflowed the display could be simply scrolled by fingertip pressure. The user 
could tune to a desired program by touching the appropriate grid section in a definitive 
way. For example, the first touch could highlight the grid box 50 as shown in Figure 9 
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and the second touch could select it. A tentative selection could be previewed locally (a^ 
a live video inset) on the controller before it was selected for viewing on the large public 
display. 

Many extra information features could be built into a service supplying the 
5 lap-top controller. For example, selecting a particular movie from a Video Dialtone 
network could initiate the display of a movie trailer on the lap-top controller screen 

In a scenario extrapolated from the growth of today's Picture-In-Piciure 
television capabilities, the lap-top controller 10 can simultaneously display several TV 
channels as shown in Figure 10. Live video or freeze-frame-sampled selections could be 
10 used. The user can switch to a program of interest by simply touching the desired 
displayed channel image. 

Advertisers might use the lap-remote to display advertisements without the 
need to disrupt prograniming on the public display (of course this would depend on 
whether they could be certain that their Ads would be viewed at all). 
^5 The lap-top controller 10 provides an excellent interface to multimedia games, 

especially those where there is a notion of a private games area and a public one. Recent 
work at Bellcore led to an experimental research prototype of a multimedia, multi-player 
electronic card game environment. Players of an electronic card game used an electronic 
platform which supported an application simulating a card deck that could be dealt into 
20 private areas and then played on public areas of linked CRT screens. 

The lap-top controller system supports true, physically separate, public and 
private areas. Service providers could construct realistic and compelling games in which 
players maintained cards or tokens in truly private areas, and played them on simulated 
(public display) tables or game boards. With secure network technology, one could 
25 support gambling and games of chance. 

Action games such as a "Tank War ' game could be supported by showing the 
common battle terrain on the public display while keeping the users' controls and gauges 
(e.g. fuel level) private on the lap-top controller screen. 

In a tele-teaching scenario, each student might have a lap-top controller placed 
30 on a desktop. Private spaces would let each work at their own pace, possibly on different 
assignments. The instructor could have access to all of the individual private spaces on 
his/her own private display, either automatically or when granted by a student. Students 
could share work by simply sliding it onto the public board. The entire object-oriented 
student environment could be available to the instructor so he or she could examine the 
35 history of the student's work and offer aid when needed. 
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Most of the lap-top controller devices described here can be assembled from 
existing subsystems. The envisioned computing power is within the range of current 
lap-top computers. Although display technology is not yet able to provide the envisioned 
wide screen color display at a moderate price, the popularity of lap-top computers and 
5 other personal electronic devices is driving the market in that direction. Low cost touch 
screen technology is already available. 

Low cost consumer devices are available that use RP links to transmit 
television signals from one VCR to several TV sets within the home. These can form the 
basis for the RF links discussed herein. In addition, digital IR linked controllers have 
10 become nearly ubiquitous devices in many homes. They provide wireless control of even 
low cost electronic devices. Their technology would be suited to the lap-top controller 
A major concern for any non-tethered device is sufficient battery capacity at 
low cost. Portable camcorders and cellular telephones are driving this market. This 
application would also be well matched to the capabilities of Bellcore's patented lithium 
15 ion battery technology, and in any event, the controller can always be returned to a 
charging cradle when not being used. 

CONCLUSION 

I have described a lap-top controller, a new interface for accessing and 
controlling FSN and ultimately Nil services. This interface (in conjunction with a large. 

20 shared display satisfies the conflicting requirements of large displays for entertainment 
along with the need for closeup, high-resolution displays for the presentation of graphics 
and text- The display combination is especially useful for information sharing 
applications and where applications require some notion of public and private access to 
multimedia information. A number of applications ranging from better access to 

25 television programming to home shopping have been disclosed. Finally, the above- 
described embodiments of the invention are intended to be illustrative only. Numerous 
alternative embodiments of the present invention may be devised without departure from 
the spirit and scope of the following claims. 
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What is claimed is: 

1 1 . A system for allowing transfer of information between multiple users, said 

2 system comprising 

3 a large display device, 

4 a plurality of individual controllers, each of said individual controllers having a 

5 display surface which defines a shared public display area and a private work display 

6 area, and 

7 a set-top box which communicates with each of said individual controllers and 

8 with said large display device. 

1 2. A system m accordance with claim I wherein said set set-top box has 

2 wireless communication with said individual controllers and wire connection to said 

3 display device. 

1 3. A system m accordance with claim 2 wherein said wireless communication 

2 comprises an IR video link and an RF data link between said set-top box and each of said 

3 controllers. 

1 4. A system in accordance with claim 3 wherein each of said individual 

2 controllers further comprises 

3 an IR transceiver, 

4 an RF transceiver, and 

5 a central processor. 

1 5. A system in accordance with claim 4 wherein each of said individual 

2 controllers further comprises 

3 a touch screen over the surface of said display surface. 

6. A system m accordance with claim 1 wherein said set-top box provides 
interconnection to an external communication network. 

7. A method for transferring information between individual controllers and a 
common display device comprising the steps of: 

assembling information in a private display area of an individual controller and 
transferring said information from said private display area of sid controller to 
5 said common display device. 

1 8. The method in accordance with claim 7 wherein said individual controller 

2 has a display surface and a touch screen over said display surface, said step of 
transferring comprising touch dragging the information from said private display area to 



3 



4 said public shared display area. 



1 



9. The method in accordance with claim 7 wherein said step of forwarding 
comprises: 



1 I 
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3 communicating by wireless from said individual controller to a set-top box and 

4 communicating by a wired connection from said set-lop box to said common display 

5 device. 
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FIG. 3 
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